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PREFACE 

At the University of Michigan School of Education we place great 
value upon research and service, in addition to instruction. This 
monograph is a result of some of the research and service which we 
hope will contribute to the betterment of American education. 

I believe Professor Robert Blackburn and doctoral student Paul 
Lingenfelter have reached some significant conclusions in this study. 
Both men are ass.jciated with the School of Education Center for the 
Study of Higher Education, which has made a number of important 
researches into the nature of higher education. 

The study described in this monograph concerns the evaluation of 
doctoral programs and was commissioned for the New York Board of Regents, 
who have been examining the rate of production of doctoral students in 
that state. I believe that aspects of the study presented in this 
monograph can have application to the examination of doctoral programs 
elsewhere . 

This study is one in a long series conducted by the School of 
Education over its more than fifty-year history. We look forward to 
continuing to contribute in a variety of areas of research. 

Wilbur J. Cohen 
Dean 



PROLOGUE 



A desire for highest quality seemingly needs no defense, iri-e>;v : i'. . 
of the product or craft involved. Yet, the consideration of "quality -r. 
any setting seems to provoke controversy. Whenever evaluative jadgrr.ents 
are made and a rank order is established, someone is offended. Hence, 
the evaluator, sometimes correctly, sometimes in error, is likely to be 
charged with elitism, bias, racism, or a host of other unsavory traits. 

Perhaps a part of the controversy regarding "quality" traces to 
ambiguities within the concept itself. After all, as Cartter (1966 :M 
notes, "in an operational sense, quality is someone's subjective assess- 
ment, for there is no way of objectively measuring what is in essence an 

attribute of value." 

The following analysis avoids the use of "quality" in the general 
sense of the word. Although "quality" for any individual may be n>ore than 
the sum of its parts, coumunication is facilitated and controversy is 
mitigated when "quality" is defined in terms of its component parts. 
Therefore, we have focused upon "excellence," defined or operationalized as 
dimensions of "quality." The constellation of "excellent" traits necessary 
for "quality" can be selected by the evaluator. However, he may find it 
more useful to let excellence stand on its own merits without striving 
to specify the more elusive characteristic, "quality." If the objectives 
of a specific program are defined, an assessment of its quality principally 
considers the degree to which those objectives are attained -- the degree 
to which excellence obtains along specific dimensions. 
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Two related matters. One is that there is agreement chat qualitv 
and excellence exist, that some programs and products have more it chan 
do others. 

Second, quality and excellence do matter. Further, they matter m 
important ways. 

Studies show unequivocally that faculty behavior is associated winh 
institutional quality (e.g., Wilson, 1942; Parsons and Piatt, 1968), thai: 
institutions differ on scales of excellence (e.g., Berelson, 1960), and on 
numerous other characteristics, as the analysis which follows shows. 

Also, despite the controversy which sometimes accompanies assessment, 
systematic evaluation of doctoral programs is becoming increasingly 
important and useful. Widely accepted projections of Ph.D. supply and 
demand (Erode, 1971; Cartter, 1971; National Science Foundation, 1971; 
Wolfe and Kidd, 1971) predict surplus doctorates in almost every field. 
Although these projections have drawn criticism (Letters, Science, 1971; 
Moses, 1972) (economic forecasting is never foolproof, especially given 
the possibility of behavioral market adjustments such as decreased appli- 
cations for doctoral study), the magnitude of the predicted surplus is 
too great to be dismissed lightly. Financial and market pressures are 
likely to force cutbacks and reallocations of resources in doctoral 
education. Careful regular assessment of excellence is necessary to 
insure that whatever actions are taken relative to doctoral programs be 
guided by a clear perception of their strengths, weaknesses, and social 
contribution. Limited resources should be invested where they can produce 
the greatest return. 
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Of course, assessment is necessary for positive reasons as we^.. 
The expansion of knowledge continues unabated, and the rates of change 
in society and in the modes of transmitting and utilizing knowledge 
are accelerating without pause. Sound research and teaching at the 
doctoral level is critical both to develop needed technological and 
social innovations and to cultivate deeper understanding of the human 
situation. Given the rate of change in the environment, regular 
reevaluation of the effectiveness of graduate education is necessary to 
forestall obsolescence and irrelevance. 

Finally, quality assessment possesses an internal virtue of no small 
consequence, one which by itself justifies the incorporation of program 
evaluation as an operational procedure as regular, say, as the annual 
audit. The assessment process gua process has salutary consequences. 
New ideas emerge; better practices are introduced, concerned self -analysis 
questions long standing assumptions, protects against dysfunctionalism, 
and, vitally important, generates a climate for healthy growth and 
development . 

For these and other concerns of government and industry and 
education, excellence in graduate schools is a sine a"£ TiSEL' Regular 
assessment, no matter how agonizing at times, is fundamental, especially 
in graduate education. How to assess quality, then, is no idle concern. 
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I. CRITICAL PROBLEMS IN THE ASSESSMENT OF EXCELLENCE 

Two problems confront the assessment of excellence in doctoral 
education: 1) Ascertaining the appropriate criteria for excellence; 
and 2) quantifying the criteria so as to permit comparisons among 
specific programs. 

The selection of criteria is by no means an easy task. It is 
plagued both by political and conceptual difficulties. For obvious 
reasons, individuals and organizations favor criteria which focus 
on their own strengths. Since institutions compete for students, 
faculty, and funds, a commonly accepted definition of excellence 
has political complexities quite aside from conceptual ones. Not- 
withstanding the political problems, even a dispassionate, disinter- 
ested observer has serious difficulty selecting a set of non-contra- 
dictory criteria. His efforts meet a barrage of conflicting evidence 
and competing arguments. 

In order to establish criteria of excellence, progran. evaluators 
first must decide what they value. From their values they then fashion 
objectives, which in turn, establish the criteria of excellence. 
Criteria vary with the objectives sought. For example, evaluators 
whose primary objective is the production of new knowledge establish 
criteria which define excellence for Ph.D. programs in terms of 
faculty and student scholarly output. They will rate a program with 
a distinguished faculty which produces much valuable research and a 
few outstanding, research-orientated Ph.D.'s much higher than evaluators 
who hold the training of college teachers as the primary objective of 



Ph.D. programs. Appropriate criteria for the second evaluatori; tnighL 
emphasize the numbers of qualified college teachers produced bv a 
program rather than the quality of faculty and student research. 

Of course, the objectives of program evaluators are rarely 
unidimensional. Furthermore, different objectives usually are 
neither mutually exclusive nor inclusive. For example, it may nor 
be possible to maximize both the generation of outstanding scholarly 
work and the production of competent college teachers. However, it 
may be possible and desirable to maintain an optimal mix by balancing 
competing values in single institutions or by seeking different 
objectives in different programs (Warren, 1967). Ideally the 
criteria established for evaluation of doctoral programs will be 
formed by a conscious review of all relevant objectives and a con- 
scious weighting of those objectives on the basis of an hierarchy of 
values. The established value hierarchy, and consequently the 
criteria for evaluating Ph.D. programs, will vary with the perceived 
needs of the institution, state or nation. Certainly it is possible 
for doctoral programs emphasizing somewhat different objectives to 
attain excellence and to receive due rewards. 

After the basic task of selecting explicit criteria has been 
completed, the second critical problem in evaluation is encountered. 
What is a valid measure of relative degrees of excellence on a stated 
criterion? For example, what is a valid measure of a faculty *r scholarly 
abilities? Is it peer evaluations? Research grants obtained? N»jmber 
of publications? 



What is a valid measure of departmental "effectiveness?" Is it 
alumni salaries? Placement of graduates? Ph.D. 's per faculty? The 
list is long. 

While most of the techniques necessary for evaluating doctoral 
programs have already been developed or can be developed relatively 
easily, problems of measurement cannot be dismissed lightly. However, 
where measurement difficulties exist, the weakness of one technique 
may be offset by the strengths of another. Hence, by utilizing 
several measures of effectiveness a relatively canprehensive and 
valid evaluation can occur. 

Each of the three elements of the evaluative process -- objectives, 
criteria, and assessment indices — have unique problems. But the 
most critical prerequisites of successful evaluation are that: 
1) Each element of the process be clearly defined; and 2) the relation- 
ships between objectives, criteria, and indices be logical and explicit. 
If these conditions are not met, the evaluative process creates 
unnecessary confusion and controversy. 

II. A CATALOG OF CRITERIA AND ASSESSMENT TECHNIQUES 

A. Reputational Studies 

The most widely known, heralded, and criticized evaluations of 
doctoral programs have been the reputational studies. Hughes (1925) 
conducted the first of these in the 20's. Similar studies have followed 
his lead. (Hughes, 1934; Keniston, 1959; Cartter, 1966; Roose and 
Anderson,. 1971).^ The two most reaent studies (1966 and 1971), sponsored 
1 Hughes (1946) published a third study of doctoral education, but it was 



not a reputational survey ♦ 



by the American Council on Education, have been methodologicailv tho •.v.ost 
sophisticated and the most widely publicized. Higgins (1968: SoS^ 
effectively sunmarizes the history of graduate school assessment in his 
unpublished rating of doctoral programs in education. 

Peer evaluation characterizes all the repuational studies. A panel 
of scholars rates the quality of an institution's faculty and /or graduate 
program in a given discipline. By combining individual panelist's 
responses, an aggregated rating of the graduate program is calculated. 

Although the criterion of excellence which dominates reputational 
studies is the scholarly ability of a faculty . the technique of peer 
rating can be used to obtain measures of other criteria. For example, 
in addition to scholarly ability the ACE studies used "program effective- 
ness" &s 6 criterion. Other criteria, such as "supportiveness of graduate 
students," "effectiveness of teaching," or "student quality" could be used. 
However, since widely scattered outside evaluators are less likely to be 
aware of conditions pertinent to such criteria than they are of a 
faculty's scholarly output, reputational assessments or other criteria can 
not be ilefined as easily as can assessments of scholarly ability. 

Ir. addition, reputational studies suffer from a lack of normative 
standards. Two matters confound peer ratings of excellence. First, 
almost by definition only a few institutions can receive "outstanding" 
ratings. Hence, what results is more a rank order of excellence rather than 
a position on a normative scale. Second, standards of excellence change 
over time. For exaitple, a university rated average today may have improved 
dramatically over the recent past while its position relative to other 
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institutions does not reflect its improvement. 

Reputational evaluations have been widely criticized (Lewis, 1968; 
Council of Graduate Schools, 1970; Bess, 1972; Elton and Rose, 1972). 
However, for what they primarily assess, scholarly competence, no better 
measure is available. Nevertheless, criticisms of reputational studies 
are not dismissed out of hand. A more complete discussion of these 
criticisms and the characteristics of the various reputational studies 
appear in Appendix A. 

Although the ACE studies rated only departments (contending that this 
is the largest unit in a university which can be accurately evaluated 
(Cartter, 1966: 106)), others have aggregated the ACE scores and published 
composite ratings of entire institutions (Ewell, 1966; Magoun, 1966). 
Magoun (1966) argues that such composite ratings are useful indices of 
effective central administration. (Several formulas for aggregating 
departmental ratings appear in Appendix B.) 

B. Some Objective Indicators of Excellence 

Several observers of graduate education have selected one or more 
objective characteristics as benchmarks of excellence. Some objective 
characteristics, labeled "correlates of quality,'* have been established 
by examining various programs rated in the ACE studies and distilling 
a cluster of objective traits associated with high quality ratings. 
Other evaluators have selected indicators on an a priori basis, defining 
quality in terms of certain characteristics and proceeding to develop an 
index of those characteristics. Calvert, et al (1971) combine several 
objective indicators to construct a general index of quality. The 
latter of these approaches is discussed first. 
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1) Scholarly Productivicy : When a critical objectivo is rho 
production of new knowledge, >scholarly output is an important criteri.^n 
of excellence. Indices of scholarly productivity have been developtKi 
for both the evaluation of individuals and departments. The typical 
index assigns weights to books, articles, reports, patents, etc., and 
a productivity score is calculated by aggregating the quantity of 
credits in each category. Examples of productivity indices can be 
found in the work of Crane (1965), Pelz and Andrews (1966), Stalling.s 
and Singhall (1971), and others. (Appendix C summarizes various schemes 
for weighting different types of output.) 

Several researchers have found a relati • iship between the most 
prolific departments and those receiving top rankings in reputational 
studies (Berelson, 1960: 127; Cartter, 1966: 80,88,101; Crane, 1965: 
703-705). Productivity, however, changes with time. An appreciable 
body of research examines faculty productivity, teaching effectiveness, 
receptivity to new ideas and adaptability as they relate to age, rank, 
and tenure. Since a number of variables affect performance, the develop 
ment of a productive faculty mix must take into account the relevant 
factors. The research is summarized in Blackburn (1972). 

An indirect, but possibly more valid index of scholarly pro- 
ductivity is the citation index used by Clark (1957), Bayer and Folger 
(1966), Cole and Cole (1967), Creager (1967), and Vyers (1970). Since 
scholarly contribution in the strictest sense of the word requires that 
other scholars utilize a piece of work, this index calculates the 
number of citations of the work of a research or departmental faculty 
which appear in subsequent scV.olarly publications. Citations of work 
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that is more than ten or fifteen years old are given extra weight, 

on the assumption that durability is an indicator of unusual quality. 

The citation method is particularly useful in the natural sciences 

since an index of citations, the Science Citation Index , is maintained 

by the Institute for Scientific Information (Garfield and Scher, 

1965). In other academic disciplines, particularly in the humanities, 

the citation technique may be less useful because citation of published 

work is not as common, and no such index of citations now exists. 

Margolis (1967) and Smith and Fieldler (1971) discuss the strengths 

and weaknesses of the citation index in greater detail. 

2) Degrees, Awards, and Other Faculty Traits: Some observers 

have used other faculty characteristics as indicator? of excellence. 

Bowker (1965) examined the number of awards won by a faculty as an 

index of its quality. Perkins and Snell (1962: 114-118) examined 

the percentage of history faculty with the Ph.D., years of experience, 

the percentage of instructors teaching graduate students, the percentage 

of faculty publishing in the past five years, the percentage publishing 

on subjects other than their Ph.D. dissertation topic, and the per- 

2 

centage of foreign specialists who have traveled abroad. Indicators 

^ The Perkins and Snell study was not primarily a rating of individual 
departments but rather a description of graduate study in history 
as a whole. However, the characteristics listed above are implicitly 
utilized as indicators of quality. 



ch as these can play a useful role in evaluating graduate departments. 
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However, they are not as useful as measures more closely related to 
the actual productive work of faculty, such as scholarly writing or 
the training of Ph.D.'s. 

3) Student Qualicy ; Another technique for evaluating doctoral 
programs has been to measure the quality of students enrolled in a 
program. In one sense this is a reputational measure, because faculties 
tend to select the best possible students and good students are attracted 
to programs with a reputation for quality. In another sense, however » 
well-qualified students are an essential element of an excellent 
program. Thus student quality can stand in its own right as a criterion 
of excellence* 

Perkins and Snell (1962: 38-39) compared history graduate 
students to students in other fields on the basis of undergraduate GPA, 
I.Q. scores, and Graduate Record Examination Scores* These indices 
can easily be used to compare students in different doctoral programs 
in the same discipline. Bowker (1965) utilized the distribution of 
Woodrow Wilson fellows among graduate programs as an index of quality* 
Cartter (1965: 107-112) used the distribution of student-attached 
fellowships (Woodrow Wilson, NSF, and others) as a cross-validation 
of the ACE study. 

Although the number of prizes obtained by the students of a 
program is an interesting and useful datum, all graduate students 
do not compete for awards. In addition, errors can occur in the 
selection process of fellowships. For example, the Ph.D. success 
rate for Woodrow Wilson Fellows is not impressive (Mooney, 1968: 52,53). 

i 

I 

k 

I 
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standardized test scores and past academic performance seem to be 
more useful indicators of student quality. Astin's (1965) study 
of undergraduate college selection can be a useful model. 

However, even standard tests will require modification to 
mitigate their cultural bias when used with minority group members. 
Several graduate programs have developed effective but unorthodox 
instruments to measure academic ability in minorities who do not 
fare well on standardized tests (Gunne and Leslie, 1972). Particularly 
since eqialized opportunity is a desirable social goal, measures of 
student ability which do not allow for cultural bias are inadequate 
indicators of quality in institutions which are enrolling large numbers 
of minority group members. Also, grade point averages are related to 
grading practices which are not uniform across the country. As colleges 
adjust high school CPA's on the basif. of experience, so do graduate 
departments weigh differentially undergraduate grading as practiced by 

different colleges. 

As yet no objective evidence exists to demonstrate relationships 
between the existence and/or percentage of faculty who are women and/or 
minority group members and the attraction and success rate of students 
from client groups heretofore discriminated against. However, limited 
experience and role theory suggest positive connections. Assuming the 
expansion of opportunities for minorities in doctoral programs is socially 
desirable, the absence or presence of minority group faculty as mentor- 
models must be considered in the assessment of a program. 

4) Ph ysical Facilities ; An obvious facility required for doctoral 
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study is an adequate library. Several writers consider library stren^tn 
an important criterion of excellence (Perkins and Snell, 1962: 137; 
Jordan, 1963; Cartter, 1965: 114-115). Jordan (1963: 374) found chac 
highly rated institutions have larger libraries and spend more per 
student on librarian salaries. 

More than a superficial review of library resources is necessary, 
however, if the library is to be used as an index of quality. A 
specific number of volumes and a budget of reasonable size are necessary 
for excellence in many disciplines, but they may not be sufficient to 
assure excellence. A massive library can be inadequately supplied in 
a given discipline. It may not be up-to-date, or it may be administered 
in a way which discourages its utilization. Only a careful review can 
determine whether a library is adequate for doctoral work. 

No systematic studies of other physical facilities for doctoral 
education (laboratories, office space, computer capabilities, seminar 
rooms, etc.) exist, possibly because these are rarely designed solely 
for graduate instruction. However, evaluation of such facilities is 
appropriate in the assessment of new or existing programs. Since the 
facilities required differ among disciplines, specialists must be 
utilized for the evaluation of physical facilities. 

5) Correlates of Reputational Quality : The National Science 
Board (1972) conducted a major study of graduate education which, 
among other things, attempted to identify the factors closely associated 
with quality graduate programs as ranked in the ACE studies. The 
following factors are found to be correlates of reputational quality: 
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1) Magnitude of the doctoral program (number of 
degrees awarded)* 

2) Amount of federal funding for academic research 
and development. 

3) Non- federal current fund income for educational 
and general purposes. 

4) Baccalaureate origins of graduate fellowship 
recipients (NSF fellowships). 

5) Baccalaureate origins of doctorates. 

6) Freshman admissions selectivity. 

7) Selection of institutions by recipients of graduate 
fellowships (NSF fellowships). 

8) Postdoctoral students in science and engineering. 

9) Doctoral awards per faculty member. 

10) Doctoral awards per graduate student. 

11) Ratio of doctorates to baccalaureate degrees. 

12) Compensation of full professors. 

13) The proportion of full professors on a faculty. 

14) Higher graduate student /faculty ratios. 

15) Departmental size of seven faculty members or more, 
(this finding is not a strict correlate calculated 
from median scores.) (National Science Board, 1969: 
49-108) (c.f. Elton and Rogers, 1971). 

(Appendix D. describes the method used to calculate correlates 
in this study.) 

Two independent kinds of data are related to the NSB correlates. 
First, the notion of mutually strong supporting disciplines (e.g., 
mathematics for physics) is shown by Berelson (1966: 124). High 
intercorrelations exist in the ratings of departments in an area 
of study (e.g., r = .76 in social sciences). Second, while no one 
debates the need for a critical mass of faculty, arguments continue 
on whether there is an optimum faculty (and hence student) size. 
Wispl (1969) and Elton and Rose (1972) report positive correlations 
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between ACE ratings and size. They imply size may be the "cause" of zr.e 
rating inasmuch as bigger is more likely to insure identification and hencx- 
insure reputation. Gallant and Prothero (1972) find no overall correlation 
between size and Roose and Anderson ratings in five selected disciplines. 
In fact, one analysis they conduct suggests an optimum size; that is, cuaiity 
drops when a certain size is exceeded. Hagstrom (1971) finds positive 
correlations with department size and the 1966 ACE ratings (.57). In fact, 
he obtains positive correlations with a number of the variables already 
mentioned — journal article output (.67), research opportunities (.45),. a 
citation index (.69), number of post doctoral fellowships (.63), as well as 
selectivity in undergraduate and Ph. D-granting institutions, awards , offices, 
and morale. 

Given the impressive list of correlates above, a temptation exists to 
assert that high value on these characteristics are necessary and sufficient 
conditions of quality doctoral education. The authors of the National Science 
Board (1969:108) report are careful to note that "the simulation of the 
values of a group of factors will not ensure a graduate program of high 
quality." That is to say, the mere existence of the correlates is not a 
sufficient condition of quality. Although high values on some of the correlates 
may be necessary for high quality (e.g., numbers 2, 12, and 15 are likely 
candidates), a causal connection is not proved by correlation analysis. 
However, calculating correlates of quality is useful for understanding the 
conditions which usually accompany good doctoral programs. Close examination 
of these programs can reveal exactly what contribution the correlated factors 
make to high quality. 

C. Efficiency as an Index of Quality 

Some definitions of excellence in doctoral education omit efficiency in any 
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form as a criterion. Others focus on efficiency and exclude many ochor 
criteria. Given the scarce resources currently available to higher educ- 
ation, efficiency is an indispetisible criterion of excellence. 

Despite conventional wisdom, the most "efficient" doctoral 
programs are not necessarily low quality diploma mills, nor are the 
least "efficient" (in degree production terms) necessarily top quality 
programs that slowly and deliberately nuture their doctoral students 
until they attain the highest standards of scholarship. The National 
Science Board evidence suggest that institutions with high reputational 
ratings produce more Ph.D.'s per graduate faculty member and per student 
enrollment than lower rated institutions (National Science Board, 1969: 
76,79). The greater efficiency of these programs may be due to economies 
of scale, support, or a host of other factors. In any evc-ut, the evidence 
indicates that "quality" and "efficiency" are not mutually exclusive. 
Just the opposite, in fact, is indicated. 

The Council of Graduate Schools has conducted a wide-ranging study 

3 

of the costs and benefits of graduate education (Powel and Lamson, 1972). 

A very useful annotated bibliography has been compiled in conjunction 
with this study. 

Their task was hampered by 1) inadequate information regarding the 
actual costs of graduate education and 2) exceedingly complex conceptual 
problems in establishing a valid measure of the social benefits of 
graduate education. The ideal quantitative analysis of the costs and 
benefits of graduate education (and many other social welfare functions) 
is beyond our capacity at this time, and the necessary techniques are 
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likely to elude us in the foreseeable future. For these reasons and 
the fact that a duplication of the CGS study is unnecessary, we make 
no effort to discuss efficiency on the broad social level. Rather, 
we concentrate upon methods to assess the relative efficiency of similar 
Ph.D. produci- j educational programs. 

In the early sixties, when gloomy prognostications of Ph.D. shortages 
were widely accepted, several studies examined the factors which contributed 
to attrition and to the inordinately long time required by some students 
to complete their work. Using enrollment data and questionnaires from 
deans, faculty, current students, successful Ph.D.'s, and drop-outs, 
these studies found that the humsnitics, followed by the social sciences, 
consistently required longer periods of study for the doctorate and 
experienced higher rates of attrition (Berelscn, 1960: 157; Wilson, 
1965: 69; Tucker, 1.969: 64-67). Breneraan has developed a theory of 
external market relationships and internal incentives to explain the 
persistence of this pattern among disciplines (Breneman, 1972). 

Other efforts to measure efficiency have focused upon the doctoral 
production rate of a faculty. Blackburn and Trowbridge (1972) examined 
the workload Implications of dissertation chairmanships and found wide 
variations in the productivity of individuals and departments. The 
National Science Board (1969) used the ratio of Ph.D.'s to graduate 
faculty and to student enrollments as measures of efficiency. Hay (1970) 
has developed an index of Ph.D. productivity which weighs the dis- 
tribution of rank in a faculty (a faculty with more full professors is 
expected to produce more Ph.D.'s), considers student enrollment (including 



the length of study in terns before the Ph.D.), and calculates pro- 
ductivity ratios which may be compared across departroenus. (.\ppendix 
E provides the details of this technique.) 

The ideal index of efficiency in Ph.D. production probably has 
not been devised. However, a good one can be developed from the 
efforts which ha"e already been made. It must include: 1) Enrollment 
data for students from the time of entry until the termination of 
their study (with or without a degree); 2) tabulations of individual 
and departmental activity relative to dissertation committees; 3) tabu- 
lation of undergraduate work loads; and 4) tabulation of all instructional 
activities (seminars, directed readings, etc.) relative to doctoral 
education. Computer-assisted analysis of such data is relatively sinple; 
gathering it in standarized form from widely scattered institutions and 
departments is more difficult. In the event that detailed data is 
available, the next best alternative is to utilize gross measures after 
the manner of the National Science Board study. 

D. Client Satisfaction Ratings 

An obvious means of evaluating an enterprise of any kind is 
to ask those it serves for their opinions. The "consumers" of doctoral 
' education are students and the employers of graduates. Both of these 
groups have been utilized in evaluative studies. 

Alciatore and Eckert (1968: U-53) asked a group of Ph.D. recipients 
to evaluate their training at the University of Minnesota. Their opinion 
of their preparation for specific tasks (teaching, research, etc.) was 
solicited as well as their general satisfaction with their career and 
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che doctoral program that launched it. A similar survey was conducco.. 
at Florida State University (1957). Such surveys of student opinion 
are most valuable when they include both current and past, successful 
and unsuccessful students so that a wide range of opinion is included 
in the study. 

Student opinion surveys have been part of several major studies 
of graduate education. Berelson (I960: 275) surveyed 3,8A3 recent 
doctoral recipients in order to gain information about their experi- 
ences and opinions about possible reforms. (H-iss, 1970: 274-303) 
conducted extensive surveys of student opinion at ten major graduate 
schools and devoted much of her study to the analysis of their 
suggestions and complaints. Tucker's (1964) study of attrition, 
Gottleib's (1961) study of graduate student socialization, and several 
other research efforts have used graduate student or alumni opinions 
as explicit evaluative instruments (Hughes, 1959; Mechanic, 1962; 
Darlington, 1970; Toombs, 1971). Harvey (1972) reviews most of the 
work on graduate students. 



^ Hall (1971) provides an excellent annotated bibliography on graduate 
education in general which includes a number of student studies. 

The other major "clients" of doctoral programs are the employers 
of graduates. McGrath (1961: 11-16) used a panel of liberal arts 
college presidents to assess the performance of graduate schools in 
ths production of college professors. About 75% of the presidents felt 
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that new Ph.D^'s came to their institutions without knowledge anc skiil^ 
important to their job. Similar surveys could obtain evaluations frcrn 
all employers of Ph.D. 's government, business, and all levels of 
education — and relate them to specific doctoral programs. 

E. Intra-Institutional Evaluation 

During the past few years many graduate schools have instituceo 
formal evaluations of academic departments. (Appendix F includes 
information about several of these.) Most review efforts utilize a 
COTmittee of administrators, outside faculty, and/or students as the 
primary review body. In many cases their function is primarily to 
provide feedback to a department in order to encourage improvement. 
However, at Minnesota and Illinois major evaluation efforts have 
been undertaken in a climate of fiscal emergency. 

Review committees generally have access to rudimentary statistical 
information and the indicators of excellence mentioned above, but 
the unique instrument for evaluation at their disposal is wide- 
ranging observations and interviews with faculty and students within 
a program. Less tangible indicators, e.g., student and faculty morale 
or teaching effectiveness, can be assessed these ways. Carefully 
corroborated impressions gathered from such a review can anticipate 
problems in seemingly strong programs or detect promise in a developing 
department. Although such techniques require funding, they can be 
used selectively to evaluate programs which are neither unusually good 
nor bad on standarized indicators. 

Review coomittee evaluations can be generated by a state agency 



L8 



as v;ell as within an institution, or h state board can merely ask 
for the results of internal self -studies as part of its total evaluation 
program. The latter approach may be ill-advised, however, since thi 
possibility of outside sanctions can discourage candor and thoroughness 
in institutional self -studies. Assured confidentiality enhances candid 
self-evaluation, reduces defensiveness, and provides a powerful impetus 
for improvement. Since intra-institucional evaluation obtains these 
advantages, an external assessi.^mt program must take care not to vitiate 
its effectiveness. 

p. Evaluation of Alumni 

The new Fh.D. is the basic output of a doctoral program. Some 
means of assessing the quality of these persons is essential to an 
evaluation program. Field ler and Biglan (1959 > examined the placement 
of new Ph.D. 's in their first Job h*i an indicator of quality. This 
approach tests a program's reputat,on in the job market. After employers 
are rated as to desirability, the placement survey is used to assess 
the competitive position of a program vis "a vis similar programs. The 
technique also can be used to assess the appropriateness of academic 
preparation to the work actually being performed by new Ph.D. *s. 

In addition to placemeuw .lurveys and the surveys of employer 
and recipient satisfaction mentlcned previously, Ph.D. alumni may be 
evaluated on the basis of their schoiorly productivity using the 
techniques listed above. Crane (1965: 704, 705) and Berelson (1960: 
126) found that the most productive scholars tend to be alumni of 
prestigious prograns. Another forthright 'leasure of Ph.D. quality 
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is content analysis of dissertations. This has not been done systematically, 
but expert outside readers can provide an unbiased estimate of quality 
by reviewing student dissertations or even dissertation abstract?. 
An example of a content analysis study is Persell (1972). 

III. NECESSARY CONDITIONS FOR ADEQUATE DOCTORAL PROGRAMS 
Professional opinion often argues that certain conditions must 
be met in order to have an adequate doctoral program. These conditions 
most often include coverage of certain specialties within a discipline 
(e.g., both organic and inorganic in chemistry) as well as the presence 
of good supportive departments in related disciplines (e.g., mathematics 
for physics). But because necessary conditions of quality are difficult 
to establish empirically, competent professionals occasionally disagree 
on the precise definition of what is absolutely essential. 

This issue is not as critical when evaluating an existing program. 
Then past productivity can be examined. However, essential conditions 
of excellence must be considered when evaluating a new program proposal. 
With trained advice to establish minimum levels for specific disciplines, 
the following are essential: 

1) Are adequate resources available to teach doctoral students? 

2) Is the dep-. jnental faculty large enough to provide 
necessary stimulation from a variety of perspectives? 

3) Are essential sub-fields covered b:, the faculty? 

4) Are the library and physical resources adequate? 

5) Do the scholarly achievements of the faculty indicate 

that they can handle advanced instruction in the proposes area 

6) Is instruction in supporting disciplines available and adequac 

7) Is there an available supply of good students? 



The guidelines for advanced studies published by the North 
« 

Central Association (1970) contain a discussion of these questions. 

IV. A MODEL FOR THE EVALUATION OF DOCTORAL EDUCATION 

The above catalogue of evaluative techniques is useful only 
as individual indicators of excellence find their appropriate place 
in an integrated conceptualization of doctoral education. Figure 1 
graphically displays such a conceptualization and the place of 
individual indicators of excellence in the broader system. 

[ Insert Figure 1] 

Although input, process, and output are interrelated, it is 
important to maintain distinctions between these categories when 
evaluating doctoral programs. A program with all of the requisite 
input values can be weak if its faculty is torn by dissension or demoralizes 
students with ambiguous requirements and debilitating criticism. Or 
a department can conceivably rank well on input and process measures 
and yet produce output of little social value. For example, an 
evaluation of a highly productive program in Slavic Languages might 
indicate the desirability of a cutback simply because there are few 
jobs for Ph.D.'s in Slavic Languages. 

At the state level, decision-makers must decide what it is they 
want from doctoral education. After this critical decision is reached, 
an existing program can be evaluated by examining its output and 
assessing the efficiency of its processes. When a proposed program is 
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examined for accreditation or an existing program tails to meet: 
the desired objectives, a careful assessment of both inputs and procc^es 
is necessary for diagnosis. Whether the prescription is curative or 
euthanasic will depend upon the cose of vitalizing the program and 
some assessment of the social need or demand for its services. 
Neither the infusion of additional resources nor euthanasia should 
be prescribed until a careful assessment is made of all aspects of a 
doctoral program. 
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APPENDIX A: A Critique of Reputational Studies 

Although a systematic criticism of reputational studies has not been 
published, attacks on this technique generally criticize the impartial irv 
of the evaluating panel, the subjectivity of reputation, the cccipetence 
of the panel, or the validity of the criteria used in reputational studie?. 
Although these criticisms do not apply with equal force to the most 
recent reputational studies, each criticism and a defense of reputational 
methodology will be considered below. 

First, the question of panel Impartiality. It is a well-kr.own fact 
that a handful of doctoral programs are alma mater to the majority of aU 
Ph D.'s. Although this pattern is changing, in 1934, Hughes top fifteen 
institutions accounted for 59% of all doctorates and in 1957 Keniston s 
top fifteen produced 43% (Berelson, I960: 97). At least one reviewer 
equates size with quality (Eells, 1957), but the high correlation between 
size and high reputational ratings may be explained in part by rater bias 
(see Wisp^, 1969; Elton and Rose, 1972). It is understandable that a 
professor would favor the department in which he is teaching or froo which 
he received his degree in a reputational survey. Consequently, departments 
with a large faculty and alumni group will receive higher ratings, so the 
argument goes. 

Other sources of bias may come from the procedure used to select evalu- 
ators. Keniston used department chairmen, and Hughes used his faculty at 
Miami University (1925) and the secretary of national scholarly associations 
(1934) to select his panel. Department chairmen may not be representative 
of professional opinion in a discipline, and relying upon a J^^gl^J^P"*^ 
to select a panel risks contamination from the biases of that individual. 

The ACE studies reduced the possibility of rater bias by systematically 
selecting a balanced panel of evaluators. Graduate deans at all institutions 
included in the evaluation were asked to select both junior and senior 
Lculty to participate on the panel. The large number of deans participating 
and the balanced distribution of rank in the panel largely eliminates the 
possibility of systematic bias through panel sdection procedures. 

The possibility of bias through current or past affiliations is less 
easily mitigated, but in a large study it is relatively """"P°^^^"^^ . ^f/'^'^'^'^ 
careful internal analysis of his data revealed that, as expected raters 
tend to be biased toward their alma mater and current e'f'Pj°y^'^ ' '"f , 

biases tend to have negligible effect. The high split-ha f "J-^^^^^^jy^^ 
Cartter's ratings suggest that systematic bias in the evaluating panel has 
been prevented by caffful techniques of panel selection and by utilizing a 
large number of respondents. 

The socond major argument against reputational studies is the intrinsic 
subjectivity of reputation. Quoting Dr. Johnson, one respondent to the 
Cartter survey observed that "a compendium of gossip is still gossip 
(C r r96L8). in a similar vein, sane argue that "-^J"^^ /,^/3""'^- 
ional 8tudies are inferior to evaluations based upon "objective traits 
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,such as faculty pablicat ions , library facilities, ecc. (Lewi,^, 19o8). A,^ 
Cartter noted, however, "objective'* criteria are based upon subjective 
notions of quality one step removed (Cartter, 1966: 4). Any effort ac 
evaluation requires a degree of trust in the subjective wisdom of the 
evaluations, whether they assess a program directly or merely establish 
ocher indicators as criteria of quality. (In defense of reputacional 
studies it should be noted that high ratings tend to be correlated with 
"objective" measures (Cartter, 1966). 

Third, some contend that a valid assessment of a doctoral program 
can only be made from first hand exposure to the program. (Council of 
Graduate Schools, 1970: 129-130). Since only a fraction of a survey panel 
is likely to have had direct exposure to more than a small fraction of the 
total population of programs, their assessments, perforce, are of dubious 
validity. This argument holds more weight when considering ratings of all 
aspects of a program than when considering ratings of a single attribute 
such as the scholarly ability of a faculty. Particularly in this case, 
faculty in a discipline have firsthand exposure to the scholarly work of 
their peers all over the country. The competence of scholars to evaluate 
the scholarly work of others is almost self-evident, even though their 
competence to assess other aspects of doctoral programs may be questioned 
if they lack firsthand information. 

Fourth, and finally, the criteria of quality used in reputational 
studies may be challenged. The ACE studies used the 'E quality of th£ graduate 
faculty (defined in terms of scholarly achievements) in your field'' and 
the overall "ef f ectivenessS of the doctoral program" as criteria for the 



5"Ef fectiveness" is defined as "the accessibility of faculty and their 
scholarly competence, curricula, educational and research facilities, the 
quality of graduate students, and other (relevant) factors." 



ratings (Cartter, 1966: 127). Since the first criterion is the most widely 
cited in reviews of the reputational studies, many argue that scholarly 
ability alone is an insufficient condition for "quality." Other factors 
such as teaching effectiveness and efficiency are important as well. The 
second criterion, as defined, encompasses all other relevant variables, but 
as mentioned above, the fact that most evaluators lack direct contact with 
most doctoral programs casts doubt upon the validity of ratings in this 
category.^ Nevertheless, scholarly competence is a necessary, if not 



^The high correlation between "scholarly achievements" and "effectiveness" 
ratings suggests that raters chose to emphasize the trait they knew best 
when evaluating "effectiveness." 



sufficient, condition of excellence and the ability of reputational surveys to 
measure this attribute has not been sufficiently challenged. 



One other factor related to criterion validity deserves mention. In a 
reputational study the criteria deemed important by the raters must be accepte 
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imolicitly. Inmost cases where professional consensus obtains 
^^ofessional observer is. and. should be. inclined to accept the judgmenc 
of the professional panel. But. if a discipline is undergoing unda^cnt a 1 
changes'in its methodology or theoretical frameworks, or if a d-cipline 
on the whole has failed to adapt itself to changing environmental need., 
the values of faculty raters may be unreliable or incongruent with those 
of a non-professional observer. 

In political science during the late fifties the controversy between 
profassors interested in behavioral approaches and those -""^"^^ 
normative structural analysis quite likely was reflected ^"^he ratings^ 
Although partisans of either perspective may have been counterbalanced by 
the o^her! interpretation of ratings in a divided discipline is risky 
without s^ne direct indication of the criteria emphasizeu by individual 
riters ^d. given the possibility that an entire discipline embraces 
Sysfun tiona values, a non-proxessional should have at least sane knowledge 
of professional norm; before accepting reputational ratings uncritically. 

TO sunmiarize. reputational studies have been criticized on the 
following grounds: 1) Panel bias; 2) Sub jecf ivity ; 3) Panel ccxnpetence. 
and 4) Criterion validity. 

in defense of reputational studies we have suggested that: 1) Pa^^l 
bias ias been largely eliminated by the careful selection procedures of the 
aJe studies 2) Subjectivity cannot be escaped in evaluation no matter what 
techniqut i^ used; 3) Professional peers are competent to evaluate scholarly 
work the central criterion in reputational studies; and 4) Although not a 
sufficient condition of general excellence, scholarly ability is necessary 
for a good doctoral program. 

Reputational studies cannot provide a comprehensive f 
n„«Hfv and thev deserve critical examination whenever used, but for ratings 
rtir;cholarry'abirity of a faculty they are a valuable and valid instrument 
of assessment. 
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APPENDIX B: Formulas for Aggi-egating Inst itut lonai 
Scores from the ACE Ratings 



Magoun (1966) calculated institutional ratings in the broad divl^l^n^ 
of study (humanities, social sciences, etc.) by adding the ACE scores ei.er 
to departments in a single institution. For example, the rating of an 
institution's social science departments on the ACE scale of 0-5.0 would be 
added together to create a total score. Exact ratings are given by the 
ACE for scores in the 3.0 - 5.0 range, and Magoun scored departments placed 
in the ''good" or ''adequate plus" categories 2.0 and 1.0 respectively. 
Lower categories were scored .50. A divisional s:ore or institutional 
score equals the total scores of its departments. 

Ewell's (1966) scoring scheme assigned points to departments based upon 
its rank order and its general classification in thv- ACE study. Departments 
in the distinguished-strong groups were given 14 poiits for making that 
category plus their inverse rank in the category. It. a field of 17 distingu- 
ished or strong departments the top ranking uepartmenr would be assigned 31 
points, the second ranking 30 points, and the 17th ran.cing department would 
receive 15 points. Departments rated "good" received 10 points, and those 
rated adequate received 5 points. Institutional scores are calculated by 
adding the scores assigned to individual departments. 

The McCurdy methodology was devised to assist the AAU in ranking 
prospective member institutions. McCurdy examined only the traditional 
liberal arts and science departments and assigned points by the category of 
evaluation received in the 1969 ACE study. Both the quality of the graduate 
faculty and the effectiveness of the graduate program ratings were 
included. Table 1 outlines the McCurdy formula (Fawcett, 1971). 

All of the above formulas, except McCurdy' s, work to the advantage of 
large institutions with many specialized departments. An institution with 
25 adequate or good departments could receive a higher scow, than an 
institution with six or seven departments all of which were rated good, 
strong, or distinguished. In an effort to correct for this bias, Wallis, 
(1971) devised a technique which compared two institutions only on the 
basis of departments they have in common. The advantage of this technique 
is that it controls for size; the disadvantage is that a separate score 
must be calculated for each institution in the comparison group. (E.g., 
Michigan has one score relative to Columbia, another relative to Yale, 
another relative to Berkeley, etc.) Wallis' formula is: 

of departments in which A is rated high er than BH- # of ^es^ 
(# of departments in which B is rated higher than A+ # of ties) 
A simpler approach to controlling for si::e in institutioual ratings is to 
add individual department scores (using «:he Magoun, EweLl, or McCurdy 
methods) and divide by the number of departments in an institution to obtain 
an average quality score. 

The various sctiemes for aggregating departmental ratings generally do 
not produce radically different rank orderings in institutions. If standard 
errors of measurement are considered, whatever differences do obtain are 
likely to be insignificant. 
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Table 1: Revised McCurdy Points Methodology 

1. Institutions listed in the Pi stinRuished-Strong Category ... 5 Poir.t. 

2. institution's Departments (Fields of Discipline) 

Rated bv OuaH t y of Gradua te Faculty (Accoraing to 
1969 A.C.E. Report) 

..... Points 

a. nistineuished Rating / ' \ 

(The Top 5 schools, including ties, in each 
Department) 

below the Distinguished Rating assuiiisd as Top 5) 

2 ?oi^.:- 
c. Good Rating • 

■(Between 2.0 & 2.9 in '69 A.C.E) 

. . . i Point 

d. Adequate Plus Rating 

(Between 1.0 & 1.9 in A.C.E.) 

3 Institution's Departments (Fields of Discipline) 
Rated bv effectiveness nraduate Program 
(According to 1969 A.C.E. Report) 

3 Points 

a. Extremely Attractive • 

(Between 2.0 & 3.0 in 1969 A.C.E.) 

2 ?oir.t> 

b . Attractive * * 

(Between 1.5 & 1.9 in 1969 A.C.E.) 

1 Pcir.-. 

c. Acceptable Plus • 

"(Between 0.8 & 1.4 in 1969 A.C.E.) 

4. T.^»^ Points Summed for each institution and ranked accordingly 



APPENDIX C: Formula.s to Weigh Books an^. Avciwio.- 
in Scholarly Production Indices 



A. Manis (1907) 

1 book =18 articles 

B. Crane (1965) 

Publications were classitied as either major or minor and tabula. ^-v, 
separately. A major publication was a book, a journal article of tr»orc 
than fifty pages, or four closely related journal articles. Minor 
publications were edited books, translations, jointly authored (more 
than three authors) books, research articles, and lab manuals. Short 
reports, book reviews, unpublished papers, and popular articles were 
not counted at all. If a professor had published a major work, his 
minor writings were not tabulated (Crane, 1965: 701-702). 

C. Cartter (1966) 



1 theoretical or research book = 6 articles 

1 textbook = 3 articles 

1 edited collection = 2 articles 

4 short communications = 1 article 

8 book reviews = 1 article 

16 book notes = 1 article 

D. Stallings and Singhal (1969) 

PUBLICATION TYPE WEIGHTED POINTS 

Book (Author) 9 
Book (Co-Author) 6 
Dissertation 5 
Book (Editor) A 
Article (Author) 3 
Article (Co-Author) 2 
Technical Report 3 
Technical Report (Co-Author) 2 
Book Review 2 
B 'c Review (Co-Author) 1 



E. Lazarsfeld and Thielens (1958: 402-407) suggest that the rank ordering 
01 professors' productivity is affected only slightly by the formula used 
to weigh books and articles. Hence, a formula which assigns equal weight 
to all forms of publication should be completely adequate for most purposes 
of comparison. This is confirmed by Cartter' s experimentation with various 
weighting schemes (Cartter, 1966: 81). 
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APPENDIX D: The National Science Board Method 
APPENDIX u calculat ing Correlates of QuaUty 

Standard correlation techniques were not used to distill the^^^^^ 

'•correlates of quality" in the N^*^-"-^"^^^/^ aggregated into 
the 1966 ACE rankings of graduate programs ,f 
institutional scores and ins..tut.ons „^ 

quality classes. '^/^ ' °^tors were calculated for each quality 
Median scores on a number of ^/f d^g^^nded uniformly (one or t 

class. If the median value °f ^"^^^"^agg a to class G, the f 

exceptions were occasionally admitted) from class A 
was deemed a quality correlate. 

, ^y.^ Hi<;fribution of median ratios of graduate fellow- 
For example, the "^;°;3°' q^aUty was A. .360; B. .296; 

ship awards to number °f •PPl^""' ! ^97. ^^ce thfse median values 
C. .273; D, .267; E. -^^5. ^' -^^^^^ss ratios of fellowship 

descend uniformly by quality '=1««S;/^^ ^he distribution 

applicants is considered a ""/^f/^^fi/^r egular or not uniformly 
drs:::drg!?he"ch:ra:trrlsririrfufsiLn woSld have been rejected a. a 
quality correlate. 

The advantage of this technique is that it^si^pUfies^the^^^^^^^ 
presentation of the evidence by displaying ^i^arlv sh ow that as 

Lsily understood In'Z correlated factor. A 

"quality" decreases so do the median^ ^^^^^^ statistical methods 
disadvantage is that by '^^P*'^""^ lost. Since no correl- 

potentially strengthening ^'^/f ef dence i^ .^^ 

ation coefficients are ^^^P^ :J;,rutror:ithi: Tquality class can only 
Moreover, the shape of the discrio ^ theoretically 

be guessed (only median values are reported ^.^^ ^^^^ 

possible for actual correlation ^^^"^^"^ „ fg^ton used in the study, 
'low levels ^^ile satisfying the test of correlation^ ^^^^^.^^ 

It is possible that the National Science B ^ deviating 

standard approaches because it removes the effects g^ ^J^^ ^^^^^^ 

been reported as well. 
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APPENDIX F: Intr«-instttutlon«l Evaluation Programs 

Following are selected pages from a descripcion of the internal 
evaluation progr«n at the University of California. Berkeley Extens .v. 
int-rnal evaluation programs are being developed in many instxtut ion. . 
Minnesota and Illinois have launched assessment programs, description^ 
of which should be available through the graduate dean s office at the^. 
institutions. Also, a major assessment program is .^^ 
Michigan. In several months detailed descriptions of this efiort shouic. 

be available. 
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l nivcrsity of Caafornia, BerkeU-y 
PR0CI:DUUES for review of doctoral l»UOCli.\MS 



The reviews of doctoral programs will be guided by ivco maior c ^nv.v.. 
that the improvement of graduate education can be fostered most effcc-.neh . > 
collecting as much information as possible about each program unaer review. ..r.>. 
ihat the same procedures will be applied to all programs rcvicv.'cd. in accorv: w.... 
these principles, the following procedures will be used as a means lor .sccunng ir.c 
types of information listed in the attached itemization. 

In each case, a principal source of information will be the members of iI-a 
department under review; other sources of information are designated in the aer-i/..:ion. 
It is anticipated that two forms of information will be solicited from depurtmem nu-inher-. 
written and oral. Written information will include various formal departmental :.nnouncc- 
ments and documents as well as less formal letters and reports. Information wu. be 
communicated orally during the course of meetings of the review panel with department 
members. 

Within each department, three subgroups may be distinguished: the chairman 
and graduate advisers; the remainder of the faculty; and the graduate studems. In 
order to p-omote candor in the communications from department member? :n tr.eso 
groui)S to tiie review panel, it is proposed that information be solicited sei^aratc.y 
from earh group. Accordingly, m addition to whatever documents arc submittec oy 
meu.bers of the three groups, three separate meetings will be scheduled lor attendance 
by the panel, one for each group. The first of these three meetings will be a dimier- 
discussion between the panel and the department chairman along with graduate advisers. 
At this time, the panel will detail Its procedures for the chairman who can also use 
the occasion to present and discuss his own views of the doctoral program in his 
department. Daytime meetings will involve the remaining two groups: faculty and 
graduate students. A final meeting will also be scheduled to which members oi ail 
three groups will be invited. This meeting will permit the panel to presem its report 
for discussion and comment by all components of the department prior to its presentation 
to the Graduate Council. Prior to preparing a final version of the report, comment on 
a preliminr^ry version will also be invited from appropriate oiilcers of other relevant 
units (e. g. Dean of the College). 

In accord with this plan, each review will be initiated by letters from the 
Chairman of the Graduate Council and the Dean of the Graduate Division to the members 
of the three departmental groups (as well as to two other pertinent groups: Teaching 
Assistants and other noa-Senate members of the instructional staff). These letters 
will invite the recipients to the appropriate meetings, outline the review procedures to 
be followed, and describe the kinds and forms of information the panel wishes to secure 
from members of each group. (Samples of these letters are enclosed). 
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The membership of the review panel will vary depending on the iden-utv o: 
the department under review. Every panel will include the Chairman ot ihe c.ra.u.i 
council and the Dean of the Graduate Division. The Council Chairman will appomt 
one or two other members of the Council to each panel, selecting persons iron- 
departments allied with that being reviewed. An Associate Dean of the Graau .te 
Division may be appointed to the panel by the Dean in view of the relat.onshi: c-.-.e 
the Associate Dean's discipline and that of the department to be reviewed. I- -.n... . . 
a graduate student may be designated to serve on the panel and to consult wun 
graduate students in the department under review. 

While the panel is securing information directly from departmental source- 
Graduate Division will take responsibility for securing and assembling all oiner in- 
formation listed in the attached itemization. In addition, the Graduate Division u il 
assist the panel in preparing its report to the Council. 



University of California, Berkeley 
GUIDELINES FOR EVALUATING DOCTORAL PROGRAMS 



Type of Information 

A. Description of Present Status and 
expected fate of disciplines in which 
work is offered 



Program 

1. Original description 

2. Approved Revisions 

3. Unauthorized Revisions 

4. Informal Description 

5. Course Offerings 

a. Content, number, variety 

b. De facto scheduling 

C, Advising 

1. Availability-frequency 

2, Quality 

D* Teaching 

1, Faculty load: formal and informal 

2. Effectiveness 

E. Evaluation of Student Progress 

1. Methods of evaluating course work 



gource of Information, 

Chairman, Advisers, FriC./.iy, ^'^uut 



Graduate Division 

Graduate Division 

Chairman and Advisers, Students 

Chairman, Advisers, Stiu ^^nts 

Catalog 

Schedule and Directory 



Chairman, advisers, other faculty 
present and former students 

Same 



Chairman, advisers, other faculty 
and schedule and directory 

Chairman, advisers, present and 
former students 



Chairman, advisers, other facult>', 
students 



*v.^Hc nf PvaUiation: Chairman, advisers, other faculty, 
e\1:.;Srrr::/er*rec«ves. 3.de„t. Oracuau D.v..o„ 

frequency, sequence, faiVire rate; 
research requirements,;thesis and 
dissertation requirements 
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3. Course offerings 

a. content, number, variety 
De facto scheduling 

F. Admissions 

1. Numbers admitted annually 

2. Procedures 

3. Criteria 



G. Resources 



1. Faculty 

a. Number filled and vacant FTE 

b. Distribution by rank 

c. Turnover rate 

d. Retirement prospects 

e. Distribution by training 
institution 

2. Studer* • ,-'ancial Support; 

TAs, KAs, Fellowships, Grants, etc. 

H. Productivity 
1. Quantity 

a. Admission rate 

b. Years to M. A. 

c. Years to Doctorate 

d. Rate of granting M. A. and 
doctorate 

e. Aldrich-Hammel index 



Source 

Graduate Division 
Chairman, advisers 
Chairman, advisers 



Chancellor 
Chancellor 
Chancellor 
Chancellor 
Chancellor 

Graduate Division 

Graduate Division: EDP 



ERIC 



Quality 

a. Sample of recent theses and 
dissertations 

b. Letters from former students 

c. Occupational history of 
former students 



Library 

Graduate Division 

Chairman, faculty, Q) 
Alumni offices 



U Morale 



1, Faculty 

2. Student 

a. Degree and methods of 

participation in departmental 
decisions 



Chairman, advisers, faculty 



Chairman, advisers, sradent 
G.A, reps. 



b. General 



Students 
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